Building Topic Models Based on Anchor Words

Suppose you were given a stack of documents, such as all of the articles published in a particular newspaper, and your goal was to make sense of this data, to determine topics that this data may be made up from. To frame this as an unsupervised learning problem, suppose the documents were written in a foreign language and came from a foreign planet. By understanding topics that these documents are about, you would be able to, given a new document, determine what characteristics it shares with other documents and uncover, ultimately, what it is about. In theory, we refer to this as the problem of unsupervised Topic Modeling, first introduced by Dave Blei et al. Of course, instead of taking news articles, this can be framed with genome sequences, audio tracks, images, and all sorts of data. The problem of Topic Modeling informally aims to discover hidden topics in documents, then annotate them according to these in order to summarize the collection of documents. This falls into the modern AI challenge of developing tools for automatic data comprehension. In the 2012 paper Learning Topic Models: Going beyond SVD [1], Sanjeev Arora, Rong Ge, and Ankur Moitra present a new method for unsupervised learning of topics, namely that of NonNegative Matrix Factorization (NMF) and provide provable bounds for the error in learning. Arora et al. motivate NMF as a more naturally derived tool for topic learning than the current approach prevailing in theory that is Singular Value Decomposition (SVD). The authors present a polynomialtime algorithm, building on their previous study on NMF [3], that similar to SVD can be realized mostly in linear algebra operations and thus achieves better running time than other local-search approaches both in theory and in practice, where the number of documents suffices to be